This section covers additional use cases that simplify the remote operation and control of the Expeditor integrator.

Platform Restart

Please, refer to the Using the IBM Expeditor integrator platform for further details (see Ref_2).

The BrowseDirectory request messages can be sent to the Expeditor integrator (CtrlQ) to retrieve a list of names of files that are currently stored in a given target (FTP or local file system) directory.

Please, refer to the Using the IBM Expeditor integrator platform for further details. (see Ref_2)

Drain and Browsing Queues

<service_name>

     <maintenance mode="ON" | "OFF">

          <param name="LogToFile" value="TRUE" | "FALSE"/>

          <param name="LogFileLocation" value="c://tmp//maintenance.log"/>

     </maintenance>

<!-- Custom Log Service -->

     <custom-log-service>

     <maintenance mode="OFF">

          <param name="LogToFile" value="TRUE|FALSE"/>

          <param name="LogFileLocation" value="c://tmp//maintenance.log"/>

     </maintenance>

...

<!-- Custom Event Service -->

     <maintenance mode="ON|OFF">

          <param name="LogToFile" value="TRUE|FALSE"/>

          <param name="LogFileLocation" value="c://tmp//maintenance.log"/>

     </maintenance>

...

<!-- Tivoli Monitoring Service - iTMS -->

     <monitoring>

          <tec-monitor name="tec1">

     <maintenance mode="ON|OFF">

          <param name="LogToFile" value="TRUE|FALSE"/>

          <param name="LogFileLocation" value="c://tmp//maintenance.log"/>

     </maintenance>

…

and is sent to the Control Queue. The default JMS Resource Adapter (see Listing 51) catches this control message and fires the

event which further triggers the Default_MaintenanceModeUpdate.flow. This flow runs the

• MaintenanceModeUpdate_RetrieveAndUpdateConfigurationAdminStore Activity to retrieve the configuration from the OSGi Configuration Admin store, changes the maintenance-mode settings (see Listing 51) according to the value provided in the Maintenance Control Message’s ResourceCmd property (ON or OFF) and creates a new XPDinteg.xml file,
• MaintenanceModeUpdate_FileWriteToFileSystem Activity to write the new XPDinteg.xml file to the file system and the
• MaintenanceModeUpdate_ConfigStoreUpdate Activity to apply the configuration changes to the OSGi Configuration Admin store.

Listing 52: Configuration of the JMS Resource Adapter for incoming Maintenance Control Messages

<adapter type="XPDINTEG_JMS_DESTINATION_ADAPTER" name="MaintenanceModeUpdate">

     <listener>

          <meta-data>MessagePurpose = 'Maintenance'</meta-data>

          <topic>com/ibm/integrator/flowtriggerevent/Maintenance/MESSAGE/JmsAdapter</topic>

     </listener>

     <configuration>

          <param name="JndiConnectionFactoryKey" value="jms/XPDinteg_ConnectionFactory"/>

          <param name="JndiDestinationKey" value="jms/XPDinteg_CtrlQ"/>

          <param name="JndiDeadletterKey" value="jms/XPDinteg_ServerDeadletterQ"/>

     </configuration>

</adapter>

...

The following chapters describe in detail the steps processed when the LocalMaintenance mode is triggered.

The Local Maintenance mode can be triggered either by a local file or a control message sent by the messaging backend.

Expeditor integrator monitors by default for the file named /config/system/maint/StartLocalMaintenance.txt

As soon as this file is created the LocalMaintanceMode flow is triggered (/flowdefs/system/Default_Start_Local_Maintenance_File.flow). The default configuration section for the Local Filesystem adapter is mentioned in Listing 53.

Listing 53: Local Filesystem Adapter for the Local Maintenance Mode trigger

<adapter type="XPDINTEG_FILE_SYSTEM_ADAPTER" name="LocalMaintenanceModeFileAdapter">

     <!-- sync mode -->

     <polling>

          <interval>10000</interval>

          <meta-data>ASCII-config/system/maint/StartLocalMaintenance.txt</meta-data>

          <topic>com/ibm/integrator/flowtriggerevent/LocalMaintenance/LocalFileSystem/LocalMaintenanceFile/LocalFileSystemAdapter/system</topic>

     </polling>

     <configuration>

          <param name="ResourceType" value="LocalFileSystemMonitorForLocalMaintance"/>

          <param name="TransferConfirmationMode" value="DELETE"/>

          <param name="DestinationName" value="config/system/maint/StartLocalMaintenance.txt"/>

          <param name="ProcessZeroLengthFiles" value="TRUE"/>

     </configuration>

</adapter>

The default configuration section for the JMS adapter that listens to this message is described in Listing 54.

Listing 54: JMS Adapter for the Local Maintenance Mode trigger

<adapter type="XPDINTEG_JMS_DESTINATION_ADAPTER" name="LocalMaintenanceModeJmsAdapter">

     <!-- async mode -->

     <listener>

          <meta-data>MessagePurpose = 'StartLocalMaintenance'</meta-data>

          <topic>com/ibm/integrator/flowtriggerevent/LocalMaintenance/MESSAGE/JmsAdapter</topic>

     </listener>

     <configuration>

          <param name="JndiConnectionFactoryKey" value="jms/XPDinteg_ConnectionFactory"/>

          <param name="JndiDestinationKey" value="jms/XPDinteg_CtrlQ"/>

          <param name="JndiDeadLetterKey" value="jms/XPDinteg_ServerDeadletterQ"/>

          <param name="ValidateLocationId" value="ON"/>

     </configuration>

</adapter>

XPDINTEG_APPLY_NEW_MICROACL Activity

Generally, this activity class allows applying a new authentication and authorization configuration to the current micro broker instance. Authenication changes are done by deploying new XPDintegMBUsers.xml file with usernames and passwords for micro broker access. Authorization changes are performed by deploying a new micro-acl.xml file which contains specific micro broker resource access definitions.

Note: This Activity operates currently with simple JAVA file objects, not with XA resources. Therefore, currently transaction safety is available.

Activity Context Parameters relevant for the Local Maintenance Flow:

• ActivateNow = true | false: specifies whether the new micro-acl.xml file is applied and activated by restarting the micro broker instance.
• NewAclFileLocation: location of new micro-acl.xml file that needs to be applied
• BackupAclFileLocation: location of old micro-acl.xml file for backup reasons (when this parameter is provided, the current/old micro-acl.xml file is backed up, otherwise it is not)
• NewMBUsersFileLocation: location of new XPDintegMBUsers.xml file that needs to be applied
• BackupMBUsersFileLocation: location of old XPDintegMBUsers.xml file for backup reasons (when this parameter is provided, the current/old XPDintegMBUsers.xml file is backed up, otherwise it is not)
• DeleteSourceFilesAfterDeployment: true | false: specifies whether the files under NewAclFileLocation and NewMBUsersFileLocation should be deleted after successful deployment

This activity is used in the Local Maintenance Flow to stop external microbroker messaging clients from sending further data to Expeditor integrator. At the same time, Expeditor integrator must still be able to continue processing already received data. This may include sending messages TO the external messaging clients. The provided ACL file /config/system/micro-acl_for_LocalMaint.xml contains a configuration which achieves this: External micro broker clients can only receive messages from Expeditor integrator (external clients are messaging clients connected to other interfaces than 127.0.0.1). Internal clients connected to 127.0.0.1 can still send to and receive messages from Expeditor integrator’s microbroker to be able to further process already received messages. The final objective in the Local Maintenance Flow is to process all already received messages and empty the local queues.

Listing 55: microbroker ACL configuration for Local Maintenance Flow (/config/system/micro-acl_for_LocalMaint.xml)

<?xml version="1.0" encoding="UTF-8"?>

<!-- IBM Property 

          micro broker ACL configuration for Expeditor integrator

          Local Maintenance Flow:

          - 1) / 2) allows only user Admin to administer broker

          - 3) all users can still connect to broker

          - 4b) internal clients on same machine (127.0.0.1) have

          read/write access to topics and queues

          - 4c) / d) external clients can only access topics and queues

          for receiving messages (no writing allowed) -->

<micro-acl policy-combination="permit-overrides"

     xmlns='http://com.ibm.micro/micro-acl'

     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

     xsi:schemaLocation="http://com.ibm.micro/micro-acl micro-acl.xsd">

     <policy rule-combination="permit-overrides">

          <resource type="broker"/>

          <!-- 1. Deny all to access resource broker -->

               <rule effect="deny"/>

          <!-- 2. Allow given users accessing resource broker for given actions, on given network interface -->

          <rule effect="permit">

          <!-- only user Admin can connect to and administer broker -->

               <subject name="Admin" />

               <action value="connect" />

               <action value="admin" />

          </rule>

          <rule effect="permit">

          <!-- 3. All users can still connect -->

               <subject name="anonymous"/>

               <action value="connect" />

          </rule>

     </policy>

     <policy rule-combination="permit-overrides">

          <resource type="topic" />

          <resource type="queue" />

          <!-- Rule_4 a): By default, deny all clients to ACCESS topic and queue resources: -->

          <rule effect="deny" />

          <!-- Rule_4 b): Only users located on same machine 127.0.0.0 can ACCESS all resources -->

          <rule effect="permit">

               <subject name="anonymous"/>

               <environment network="127.0.0.0/8"/>

          </rule>

          <!-- Rule_4 c): External users can only receive messages from topic resources -->

          <rule effect="permit">

               <subject name="anonymous"/>

               <resource type="topic"/>

               <action value="subscribe"/>

          </rule>

          <!-- Rule_4 d): External users can only receive messages from queue resources -->

          <rule effect="permit" >

               <subject name="anonymous"/>

               <resource type="queue"/>

               <action value="get"/>

          </rule>

          </policy>

</micro-acl>

Listing 56: Apply New Micro ACL Activity section for the Local Maintenance Flow

<XPDintegActivity Name="MaintenanceModeUpdate_StopExternalmbClients_withNewMicroAcl"

     ActivityName="XPDINTEG_APPLY_NEW_MICROACL"

     ActivateNow="true"

     NewAclFileLocation="config/system/micro-acl_for_LocalMaint.xml"      BackupAclFileLocation="config/backup/micro-acl_for_LocalMaint.xml.backup"

     NewMBUsersFileLocation="config/system/XPDintegMBUsers_for_LocalMaint.xml"     BackupMBUsersFileLocation="config/backup/XPDintegMBUsers_for_LocalMaint.xml.backup"

     DeleteSourceFilesAfterDeployment="false"

/>

This ACS flow activity generally allows controlling OSGi bundles during ACS flow execution. Its main purpose for Expeditor integrator is supporting starting and stopping of already nstalled bundles. The OSGi Framework BundleContext and Bundle interface are used to locate and access the bundle which needs to be controlled.

Activity Context Parameters relevant for the Local Maintenance Flow:

• BundleName = name of the controlled bundle in the platform
• Command = START | STOP

Currently, this activity is used to stop the Expeditor integrator REST Adapter Service in order to avoid possible connected clients to send further messages to integrator through this interface.

Listing 57: Bundle Control Activity section for the flow

<XPDintegActivity

     Name="MaintenanceModeUpdate_BundleControl_StopRESTAdapter"

     ActivityName="XPDINTEG_BUNDLE_CONTROL"

     BundleName="com.ibm.rcp.integrator.rest"

     Command="STOP"

/>

The XPDINTEG_RESOURCE_MONITOR_CONTROL activity is used to stop all custom resource monitors like FTP, LFS, SSH, JDBC and HTTP adapters, This activity offers two configuration parameters: "AdapterTypeList" and "Command".

The "AdapterTypeList" parameter allows all possible adapter types that are known within Expeditor integrator. For the Local Maintenance Mode it must be set to “XPDINTEG_FILE_SYSTEM_ADAPTER,XPDINTEG_FTP_ADAPTER,XPDINTEG_SSH_ADAPTER,XPDINTEG_HTTP_ADAPTER,XPDINTEG_JDBC_ADAPTER,XPDINTEG_GENERIC_ADAPTER”.

The “Command” parameter allows the values “START” or “STOP”. For the Local Maintenance Mode it must be set to “STOP”.

The Listing 58 details the whole activity section for the flow:

Listing 58: Resource Monitor Control Activity section for the flow

<XPDintegActivity

     Name="MaintenanceModeUpdate_Stop_Resource_Monitors"

     ActivityName="XPDINTEG_RESOURCE_MONITOR_CONTROL"

     AdapterTypeList="XPDINTEG_FILE_SYSTEM_ADAPTER,XPDINTEG_FTP_ADAPTER,XPDINTEG_SSH_ADAPTER,XPDINTEG_HTTP_ADAPTER,XPDINTEG_JDBC_ADAPTER,XPDINTEG_GENERIC_ADAPTER"

     Command="STOP"

/>

The XPDINTEG_MB_BRIDGE_CONTROL activity is used to stop the incoming communication from messaging backend. Therefore all incoming Bridge flows are stopped. This activity offers therefore 2 configuration parameters: “FlowTypeList” and “Command”. The “FlowTypeList” parameter must be set to “INBOUND” and the “Command” parameter must be set to “STOP”.

Listing 59 details the whole activity section for the flow.

Listing 59: Bridge Control Activity section for the flow

<XPDintegActivity

     Name="MaintenanceModeUpdate_Stop_Inbound_Messaging_Communication"

     ActivityName="XPDINTEG_MB_BRIDGE_CONTROL"

     FlowTypeList="INBOUND"

Command="STOP"

/>

The XPDINTEG_WAIT_FOR_QUEUE_PROCESSING activity is used to wait for a configurable time interval until all messages are processed and delivered e.g. to the Backend system or other external clients. The activity does the following:

• First check if all queues are empty considering an “Ignore list”, if there are no messages on the queues, the activity finishes
• if there are messages, the activity waits for a configurable time allowing to process the messages being left on the queues
• second check if all queus are empty considering an “Ignore List”, if there are no messages on the queues the activity finishes
• if there are still messages on the queues, the messages being left are processed by the “UnprocessedMessageAction”. If the action is set to “DISCARD” all messages being left are deleted, if the action is set to “REMOVE” the messages are sent to the provided queue name (“UnprocessedMessageTarget”). If this queue is bridged to a backend system, these messages are sent to this system and are not lost.
• In case the action was “REMOVE”, the activity is waiting again for the configured time interval to allow further message processing (e.g. sending the messages to backend systems)
• Final empty check is done to check if all messages are processed. If not, a WARNING logging is done and then the activity finishes.

The activity offers 4 configuration parameters:

• "UnprocessedQueueTimeout”: This parameter represents the waiting interval in ms allowing to process further messages. Optional parameter, if not set a default interval of 30 seconds is applied.
• “QueueIgnoreList”: This parameter is comma-separated list of queue names being during the empty-check procedure. Optional parameter.
• “UnprocessedMessageAction”: This parameter describes the action to be processed in case messages can not be processed within the waiting interval (UnprocessedQueueTimeout). Allowed values are “DISCARD” to delete messages not being processed and “REMOVE” to move the messages to a different queue, e.g. a queue that is bridged to the backend system.Required parameter.
• “UnprocessedMessageTarget”: This parameter describes the queue name were unprocessed messages are sent to. It is only required if the “UnprocessedMessageAction” is set to “REMOVE”.

Listing 60: Wait for Queueprocessing activity example for the flow

<XPDintegActivity

     Name="MaintenanceModeUpdate_Wait_For_Queue_Processing"

     ActivityName="XPDINTEG_WAIT_FOR_QUEUE_PROCESSING"

     UnprocessedQueueTimeout="60000"

     QueueIgnoreList="XPDinteg_LocalDeadletterQ"

     UnprocessedMessageAction="REMOVE"

     UnprocessedMessageTarget="XPDinteg_ServerDeadletterQ"

/>

• Local file system files (check whether size of defined files exceed configurable thresholds)
• Local Expeditor integrator queues (check for queues running close to their size limit and identify messages sitting in queues which do not get processed on time – exceed their time-to-live, TTL, parameter)
• System resources (check whether available RAM and disk space are sufficient)

Each monitored resources type has one assigned ACS Activity:

• HOUSE_KEEPING_LOCAL_FILE_SYSTEM_ACTIVITY,
• HOUSE_KEEPING_JMS_ACTIVITY
• HOUSE_KEEPING_DURSUBS_ACTIVITY and
• HOUSE_KEEPING_SYSTEM_RESOURCES_ACTIVITY

The HouseKeepingAdapter is either triggered by the HouseKeeping Control Message (configured through the
<adapter type="XPDINTEG_JMS_DESTINATION_ADAPTER" name="HouseKeepingAdapter1"

or the House Keeping Generic Resource Adapter
<adapter type="XPDINTEG_GENERIC_ADAPTER" name="HouseKeepingAdapter2">

by publishing the com/ibm/integrator/flowtriggerevent/HouseKeeping
event. This event triggers the Default_HouseKeeping.flow (see Figure 16). This flow runs the three configurable House Keeping Activities in sequential order:

• HouseKeeping_HouseKeepingLocalFileSystem
• HouseKeeping_HouseKeepingJMS
• HouseKeeping_HouseKeepingDurSubs
• HouseKeeping_HouseKeepingSystemResources

The House Keeping flow results in the /persistent/housekeeping_<date>.log file which lists all checked resources. This file could the be regularly picked up by a specifically configured LocalFileSystem file Resource Adapter which looks for the existence of the /persistence/housekeeping_<date>.log file and transmits it to back-end queue as configured.


Figure 16: Expeditor integrator House Keeping flow

The default House Keeping flow also includes one FileToMessageWrite Activity which sends the House Keeping report as message to the configured JndiDestinationKey (queue):

<XPDinteg Activity Name="HouseKeeping_HouseKeepingFileToMessageWrite"

     <ActivityName="XPDINTEG_FILE_TO_MESSAGE_WRITE"

     JndiConnectionFactoryKey="jms/XPDinteg_ConnectionFactory"

     JndiDestinationKey="jms/XPDinteg_FileSysToMsgQ"/>

<configuration activity="HOUSE_KEEPING_LOCAL_FILE_SYSTEM_ACTIVITY">

          <param name="my_resource_name"

               value=TType:FILE;Destination:REGEX-workspace/logs/(.)*.xml;Threshold:4096;Action:REPORT_WARNING"/>

</configuration>

Listing 61: Example for HouseKeeping files Activity

<configuration activity="HOUSE_KEEPING_LOCAL_FILE_SYSTEM_ACTIVITY">

     <param name="HouseKeepingFile.1" value="Type:FILE;

          Destination:REGEX-workspace/logs/(.)*.xml;

          Threshold:4096;

          Action:REPORT_WARNING"/>

          …

</configuration>

<configuration activity="HOUSE_KEEPING_JMS_ACTIVITY"> 

     <param name="my_resource_name" value="JNDIKey:my_queue;

          Threshold:my_queue_depth_threshold;

          ThresholdAction:REPORT_WARNING;

          TTLAction:REMOVE;

          RemoveTargetJNDI:jms/XPDinteg_ServerDeadletterQ"/>

          …

</configuration>

Listing 62 provides an example configuration in which the local MsgToFtpQueueis checked. If more then 30000 messages are in the MsgToFtpQueue when the House Keeping takes place, REPORT_WARNING is created (log message for the OSGi Console and the Custom Log Service). If messages are found with an exceeded value of the JMS Custom Header property Time-To-Live (TTL), they are moved to the ServerDeadletterQ.

Listing 62: Example configuration for HouseKeeping JMS Activity

<configuration activity="HOUSE_KEEPING_JMS_ACTIVITY">

     <param name="HouseKeepingDurSub_1"

          value="JNDIKey:jms/XPDinteg_MsgToFtpQueue;Threshold:30000;ThresholdAction:REPORT_WARNING;

          TTLAction:REMOVE; RemoveTargetJNDI:jms/XPDinteg_ServerDeadletterQ"/>

          …

</configuration>

<configuration activity="HOUSE_KEEPING_DURSUBS_ACTIVITY">

     <param name="HouseKeepingDurSub_1"

          value="DurableSubName:ALL;InactivityCounterThreshold:2;

          ThresholdAction:NONE;ThresholdInfo:REPORT_ERROR"/>

...

</configuration>

Listing 63 provides an example configuration in which the all durable subscriptions are checked. If a durable subscribed client has been disconnected during the last three housekeeping runs (InactivityCounterThreshold=2), REPORT_WARNING messages are created (log message for the OSGi Console and the Custom Log Service). If the client is still not connected, the durable subscription will be deleted.

Listing 63: Example configuration for HouseKeeping DURSUBS Activity

<configuration activity="HOUSE_KEEPING_DURSUBS_ACTIVITY">

     <param name="HouseKeepingDurSub.1"

          value="DurableSubName:ALL;InactivityCounterThreshold:2;

          ThresholdAction:DELETE;ThresholdInfo:REPORT_WARNING"/>

     ...

</configuration>

Local System Resources House Keeping

<configuration activity="HOUSE_KEEPING_SYSTEM_RESOURCES_ACTIVITY">

     <param name="my_resouce_name"

          value="Threshold:my_threshold;

          Action:REPORT_WARNING"/>

Listing 64 provides an example configuration in which

• the currently used RAM is checked and a WARNING message is created if 256 MB are exceeded,
• the currently used disk space is checked and a WARNING message is created if it exceeds 1GB,
• the current number of files of Expeditor integrator runtime is checked and a WARNING message is created if there are more then 1000.

Listing 64: Example configuration for HouseKeeping system resources Activity

<configuration activity="HOUSE_KEEPING_SYSTEM_RESOURCES_ACTIVITY">

     <param name="RAM" value="Threshold:256000;Action:REPORT_WARNING"/>

          <param name="DiskSpace" value="Threshold:1000000;Action:REPORT_WARNING"/>

          <param name="Files" value="Threshold:1000;Action:REPORT_WARNING"/>

</configuration>